Convergence of random variables

In probability theory, there exist several different notions of convergence of random variables. The convergence of sequences of random variables to some limit random variable is an important concept in probability theory, and its applications to statistics and stochastic processes. The same concepts are known in more general mathematics as stochastic convergence and they formalize the idea that a sequence of essentially random or unpredictable events can sometimes be expected to settle down into a behaviour that is essentially unchanging when items far enough into the sequence are studied. The different possible notions of convergence relate to how such a behaviour can be characterised: two readily understood behaviours are that the sequence eventually takes a constant value, and that values in the sequence continue to change but can be described by an unchanging probability distribution.

Contents

Background

"Stochastic convergence" formalizes the idea that a sequence of essentially random or unpredictable events can sometimes be expected to settle into a pattern. The pattern may for instance be

Some less obvious, more theoretical patterns could be

These other types of patterns that may arise are reflected in the different types of stochastic convergence that have been studied.

While the above discussion has related to the convergence of a single series to a limiting value, the notion of the convergence of two series towards each other is also important, but this is easily handled by studying the sequence defined as either the difference or the ratio of the two series.

For example, if the average of n uncorrelated random variables Yi, i = 1, ..., n, all having the same finite mean and variance, is given by

X_n = \frac{1}{n}\sum_{i=1}^n Y_i\,,

then as n tends to infinity, Xn converges in probability (see below) to the common mean, μ, of the random variables Yi. This result is known as the weak law of large numbers. Other forms of convergence are important in other useful theorems, including the central limit theorem.

Throughout the following, we assume that (Xn) is a sequence of random variables, and X is a random variable, and all of them are defined on the same probability space \scriptstyle (\Omega, \mathcal{F}, P ).

Convergence in distribution

Examples of convergence in distribution
Dice factory
Suppose a new dice factory has just been built. The first few dice come out quite biased, due to imperfections in the production process. The outcome from tossing any of them will follow a distribution markedly different from the desired uniform distribution.

As the factory is improved, the dice become less and less loaded, and the outcomes from tossing a newly produced die will follow the uniform distribution more and more closely.

Tossing coins
Let Xn be the fraction of heads after tossing up an unbiased coin n times. Then X1 has the Bernoulli distribution with expected value μ = 0.5 and variance σ2 = 0.25. The subsequent random variables X2, X3, … will all be distributed binomially.

As n grows larger, this distribution will gradually start to take shape more and more similar to the bell curve of the normal distribution. If we shift and rescale Xn’s appropriately, then \scriptstyle Z_n = \sqrt{n}(X_n-\mu)/\sigma will be converging in distribution to the standard normal, the result that follows from the celebrated central limit theorem.

Graphic example

Suppose { Xi } is an iid sequence of uniform U(−1,1) random variables. Let \scriptstyle Z_n = {\scriptscriptstyle\frac{1}{\sqrt{n}}}\sum_{i=1}^n X_i be their (normalized) sums. Then according to the central limit theorem, the distribution of Zn approaches the normal N(0, ⅓) distribution. This convergence is shown in the picture: as n grows larger, the shape of the pdf function gets closer and closer to the Gaussian curve.

With this mode of convergence, we increasingly expect to see the next outcome in a sequence of random experiments becoming better and better modeled by a given probability distribution.

Convergence in distribution is the weakest form of convergence, since it is implied by all other types of convergence mentioned in this article. However convergence in distribution is very frequently used in practice; most often it arises from application of the central limit theorem.

Definition

A sequence {X1X2, …} of random variables is said to converge in distribution, or converge weakly, or converge in law to a random variable X if


    \lim_{n\to\infty} F_n(x) = F(x),

for every number xR at which F is continuous. Here Fn and F are the cumulative distribution functions of random variables Xn and X correspondingly.

The requirement that only the continuity points of F should be considered is essential. For example if Xn are distributed uniformly on intervals [0, 1n], then this sequence converges in distribution to a degenerate random variable X = 0. Indeed, Fn(x) = 0 for all n when x ≤ 0, and Fn(x) = 1 for all x1n when n > 0. However, for this limiting random variable F(0) = 1, even though Fn(0) = 0 for all n. Thus the convergence of cdfs fails at the point x = 0 where F is discontinuous.

Convergence in distribution may be denoted as

\begin{align}
  & X_n \ \xrightarrow{d}\ X,\ \ 
    X_n \ \xrightarrow{\mathcal{D}}\ X,\ \ 
    X_n \ \xrightarrow{\mathcal{L}}\ X,\ \  
    X_n \ \xrightarrow{d}\ \mathcal{L}_X, \\
  & X_n \rightsquigarrow X,\ \ 
    X_n \Rightarrow X,\ \ 
    \mathcal{L}(X_n)\to\mathcal{L}(X),\\ 
  \end{align}

where \scriptstyle\mathcal{L}_X is the law (probability distribution) of X. For example if X is standard normal we can write X_n\,\xrightarrow{d}\,\mathcal{N}(0,\,1).

For random vectors {X1X2, …} ⊂ Rk the convergence in distribution is defined similarly. We say that this sequence converges in distribution to a random k-vector X if


    \lim_{n\to\infty} \operatorname{Pr}(X_n\in A) = \operatorname{Pr}(X\in A)

for every A ⊂ Rk which is a continuity set of X.

The definition of convergence in distribution may be extended from random vectors to more complex random elements in arbitrary metric spaces, and even to the “random variables” which are not measurable — a situation which occurs for example in the study of empirical processes. This is the “weak convergence of laws without laws being defined” — except asymptotically.[1]

In this case the term weak convergence is preferable (see weak convergence of measures), and we say that a sequence of random elements {Xn} converges weakly to X (denoted as Xn ⇒ X) if


    \operatorname{E}^*h(X_n) \to \operatorname{E}\,h(X)

for all continuous bounded functions h(·).[2] Here E* denotes the outer expectation, that is the expectation of a “smallest measurable function g that dominates h(Xn)”.

Properties

Convergence in probability

Examples of convergence in probability
Height of a person
Consider the following experiment. First, pick a random person in the street. Let X be his/her height, which is ex ante a random variable. Then you start asking other people to estimate this height by eye. Let Xn be the average of the first n responses. Then (provided there is no systematic error) by the law of large numbers, the sequence Xn will converge in probability to the random variable X.
Archer
Suppose a person takes a bow and starts shooting arrows at a target. Let Xn be his score in n-th shot. Initially he will be very likely to score zeros, but as the time goes and his archery skill increases, he will become more and more likely to hit the bullseye and score 10 points. After the years of practice the probability that he hit anything but 10 will be getting increasingly smaller and smaller. Thus, the sequence Xn converges in probability to X = 10.

Note that Xn does not converge almost surely however. No matter how professional the archer becomes, there will always be a small probability of making an error. Thus the sequence {Xn} will never turn stationary: there will always be non-perfect scores in it, even if they are becoming increasingly less frequent.

The basic idea behind this type of convergence is that the probability of an “unusual” outcome becomes smaller and smaller as the sequence progresses.

The concept of convergence in probability is used very often in statistics. For example, an estimator is called consistent if it converges in probability to the quantity being estimated. Convergence in probability is also the type of convergence established by the weak law of large numbers.

Definition

A sequence {Xn} of random variables converges in probability towards X if for all ε > 0


    \lim_{n\to\infty}\Pr\big(|X_n-X| \geq \varepsilon\big) = 0.

Formally, pick any ε > 0 and any δ > 0. Let Pn be the probability that Xn is outside the ball of radius ε centered at X. Then for Xn to converge in probability to X there should exist a number Nδ such that for all n ≥ Nδ the probability Pn is less than δ.

Convergence in probability is denoted by adding the letter p over an arrow indicating convergence, or using the “plim” probability limit operator:


    X_n \ \xrightarrow{p}\ X,\ \ 
    X_n \ \xrightarrow{P}\ X,\ \ 
    \underset{n\to\infty}{\operatorname{plim}}\, X_n = X.

For random elements {Xn} on a separable metric space (Sd), convergence in probability is defined similarly by[4]


    \Pr\big(d(X_n,X)\geq\varepsilon\big) \to 0, \quad \forall\varepsilon>0.

Properties

Almost sure convergence

Examples of almost sure convergence
Example 1
Consider an animal of some short-lived species. We note the exact amount of food that this animal consumes day by day. This sequence of numbers will be unpredictable in advance, but we may be quite certain that one day the number will become zero, and will stay zero forever after.
Example 2
Consider a man who starts tomorrow to toss seven coins once every morning. Each afternoon, he donates a random amount of money to a certain charity. The first time the result is all tails, however, he will stop permanently.

Let X1, X2, … be the day by day amounts the charity receives from him.

We may be almost sure that one day this amount will be zero, and stay zero forever after that.

However, when we consider any finite number of days, there is a nonzero probability the terminating condition will not occur.

This is the type of stochastic convergence that is most similar to pointwise convergence known from elementary real analysis.

Definition

To say that the sequence Xn converges almost surely or almost everywhere or with probability 1 or strongly towards X means that


    \operatorname{Pr}\!\left( \lim_{n\to\infty}\! X_n = X \right) = 1.

This means that the values of Xn approach the value of X, in the sense (see almost surely) that events for which Xn does not converge to X have probability 0. Using the probability space \scriptstyle (\Omega, \mathcal{F}, P ) and the concept of the random variable as a function from Ω to R, this is equivalent to the statement


    \operatorname{Pr}\Big( \omega \in \Omega�: \lim_{n \to \infty} X_n(\omega) = X(\omega) \Big) = 1.

Another, equivalent, way of defining almost sure convergence is as follows:


    \operatorname{Pr}\Big( \liminf \big\{\omega \in \Omega�: | X_n(\omega) - X(\omega) | < \varepsilon \big\} \Big) = 1 \quad\text{for all}\quad \varepsilon>0.

Almost sure convergence is often denoted by adding the letters a.s. over an arrow indicating convergence:


    X_n \, \xrightarrow{\mathrm{a.s.}} \, X.

For generic random elements {Xn} on a metric space (S, d), convergence almost surely is defined similarly:


    \operatorname{Pr}\Big( \omega\in\Omega:\, d\big(X_n(\omega),X(\omega)\big)\,\underset{n\to\infty}{\longrightarrow}\,0 \Big) = 1

Properties

Sure convergence

To say that the sequence or random variables (Xn) defined over the same probability space (i.e., a random process) converges surely or everywhere or pointwise towards X means

\lim_{n\rightarrow\infty}X_n(\omega)=X(\omega), \, \, \forall \omega \in \Omega.

where Ω is the sample space of the underlying probability space over which the random variables are defined.

This is the notion of pointwise convergence of sequence functions extended to sequence of random variables. (Note that random variables themselves are functions).

\big\{\omega \in \Omega \, | \, \lim_{n \to \infty}X_n(\omega) = X(\omega) \big\} = \Omega.

Sure convergence of a random variable implies all the other kinds of convergence stated above, but there is no payoff in probability theory by using sure convergence compared to using almost sure convergence. The difference between the two only exists on sets with probability zero. This is why the concept of sure convergence of random variables is very rarely used.

Convergence in mean

We say that the sequence Xn converges in the r-th mean (or in the Lr-norm) towards X, for some r ≥ 1, if r-th absolute moments of Xn and X exist, and


    \lim_{n\to\infty} \operatorname{E}\left( |X_n-X|^r \right) = 0,

where the operator E denotes the expected value. Convergence in r-th mean tells us that the expectation of the r-th power of the difference between Xn and X converges to zero.

This type of convergence is often denoted by adding the letter Lr over an arrow indicating convergence:

X_n \, \xrightarrow{L^r} \, X.

The most important cases of convergence in r-th mean are:

\underset{n \to \infty}{\operatorname{l.i.m.}} X_n = X.\,\!

Convergence in the r-th mean, for r > 0, implies convergence in probability (by Markov's inequality), while if r > s ≥ 1, convergence in r-th mean implies convergence in s-th mean. Hence, convergence in mean square implies convergence in mean.

Convergence in rth-order mean

Examples of convergence in rth-order mean.

Basic example: A newly built factory produces cans of beer. The owners want each can to contain exactly a certain amount.

Knowing the details of the current production process, engineers may compute the expected error in a newly produced can.

They are continuously improving the production process, so as time goes by, the expected error in a newly produced can tends to zero.

This example illustrates convergence in first-order mean.

This is a rather "technical" mode of convergence. We essentially compute a sequence of real numbers, one number for each random variable, and check if this sequence is convergent in the ordinary sense.

Formal definition

If \scriptstyle \lim_{n \to \infty} E(|X_n - a|^r )=0 for some real number a, then {Xn} converges in rth-order mean to a.

Commonly used notation: X_n \stackrel{L_r}{\rightarrow} a.

Properties

The chain of implications between the various notions of convergence are noted in their respective sections. They are, using the arrow notation:

\begin{matrix}
  \xrightarrow{L^s}  & \underset{s>r\geq1}{\Rightarrow} &  \xrightarrow{L^r}  &             & \\
                     &                                  &     \Downarrow      &             & \\
  \xrightarrow{a.s.} &            \Rightarrow           & \xrightarrow{\ p\ } & \Rightarrow & \xrightarrow{\ d\ }
  \end{matrix}

These properties, together with a number of other special cases, are summarized in the following list:

\sum_n \mathbb{P} \left(|X_n - X| > \varepsilon\right) < \infty,
then we say that Xn converges almost completely, or almost in probability towards X. When Xn converges almost completely towards X then it also converges almost surely to X. In other words, if Xn converges in probability to X sufficiently quickly (i.e. the above sequence of tail probabilities is summable for all ε > 0), then Xn also converges almost surely to X. This is a direct implication from the Borel-Cantelli lemma.
S_n = X_1%2B\cdots%2BX_n \,
then Sn converges almost surely if and only if Sn converges in probability.

\left. \begin{array}{ccc}
X_n\xrightarrow{a.s.} X
\\ \\
|X_n| < Y
\\ \\
\mathrm{E}(Y) < \infty
\end{array}\right\} \quad\Rightarrow \quad X_n\xrightarrow{L^1} X

See also

Notes

  1. ^ Bickel et al. 1998, A.8, page 475
  2. ^ van der Vaart & Wellner 1996, p. 4
  3. ^ Romano & Siegel 1985, Example 5.26
  4. ^ Dudley 2002, Chapter 9.2, page 287
  5. ^ Dudley 2002, p. 289
  6. ^ Porat, B. (1994). Digital Processing of Random Signals: Theory & Methods. Prentice Hall. pp. 19. ISBN 0130637513. 
  7. ^ a b c d e f van der Vaart 1998, Theorem 2.7
  8. ^ Gut, Allan (2005). Probability: A graduate course. Theorem 3.4: Springer. ISBN 0387228330. 
  9. ^ van der Vaart 1998, Th.2.19

References

  • Bickel, Peter J.; Klaassen, Chris A.J.; Ritov, Ya’acov; Wellner, Jon A. (1998). Efficient and adaptive estimation for semiparametric models. New York: Springer-Verlag. ISBN 0387984739. LCCN QA276.8.E374. 
  • Billingsley, Patrick (1986). Probability and Measure. Wiley Series in Probability and Mathematical Statistics (2nd ed.). Wiley. 
  • Billingsley, Patrick (1999). Convergence of probability measures (2nd ed.). John Wiley & Sons. pp. 1–28. ISBN 0471197459. 
  • Dudley, R.M. (2002). Real analysis and probability. Cambridge, UK: Cambridge University Press. ISBN 052180972X. 
  • Grimmett, G.R.; Stirzaker, D.R. (1992). Probability and random processes (2nd ed.). Clarendon Press, Oxford. pp. 271–285. ISBN 0-19-853665-8. 
  • Jacobsen, M. (1992). Videregående Sandsynlighedsregning (Advanced Probability Theory) (3rd ed.). HCØ-tryk, Copenhagen. pp. 18–20. ISBN 87-91180-71-6. 
  • Ledoux, Michel; Talagrand, Michel (1991). Probability in Banach spaces. Berlin: Springer-Verlag. pp. xii+480. ISBN 3-540-52013-9. MR1102015. 
  • Romano, Joseph P.; Siegel, Andrew F. (1985). Counterexamples in probability and statistics. Great Britain: Chapman & Hall. ISBN 0412989018. LCCN 1985 QA273.R58 1985. 
  • van der Vaart, Aad W.; Wellner, Jon A. (1996). Weak convergence and empirical processes. New York: Springer-Verlag. ISBN 0387946403. LCCN 1996 QA274.V33 1996. 
  • van der Vaart, Aad W. (1998). Asymptotic statistics. New York: Cambridge University Press. ISBN 9780521496032. LCCN 1998 QA276.V22 1998. 
  • Williams, D. (1991). Probability with Martingales. Cambridge University Press. ISBN 0521406056. 
  • Wong, E.; Hájek, B. (1985). Stochastic Processes in Engineering Systems. New York: Springer–Verlag. 

This article incorporates material from the Citizendium article "Stochastic convergence", which is licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License but not under the GFDL.